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ABSTBACT 

This paper reports findings of research done 
concerning the productivity of types of Classroom reading instruction 
in a sample of Nev York State elementary classroom^. Data vas 
gathered from approximately 5,800 foAirth, fifth, and sixth grade 
students in five Bew York State school districts during the spring 
semester of the 197a-75 school year. The main design of the analysis 
is based upon the "production function" para'digjn used by economi/sts, 
in which attenrtion is paid^to how the inputs of "ingredients" to a 
production process are mixed together to produce outcomes. To 
pinpoint instructional /resources going into reading, teachers, aides,' 
and specialists were asked to supply figures for minutes per week of 
* reading 'instruction in each of four instructional modes: whole group, 
small group, individualized instruction, and individual help. 
Oiitcomes were measured by a standardized achievement test and a set 
of criterion-referenced tests. An important finding was that minutes 
o»f classroom teacher instruction seem to be related to stud,ent 
performances when measured by the criterion-ref^^enced tests but not 
whfen measured by the staijdardized test. (MKM) 
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This paper reports findings wllth respect to the productivity of types 
of classroom reading instruction in a large ^ainple of New York State ele- 
mentary school classrocpis. The main design of>the analysis is based upon^ 
the "production function" paradigm used by economists, in which attention 
is paid to how the inpiAs or "ingredients" to a production process are 

mixed together to produce outcom&s. The resulting approach for the 

' , / i 

analysis in this study^waf similar to that suggested in recent years by 
some educational writejfts concerning the desirability of looking into 'the 
returns to instructional "time". Besides using a traditional standardized 
achievement test to m^asux^ outcomes, the study also utilized a set of 
criterion-referenced ^fests that have been developed by the New York State 
Education Dei>6urtment . ,: This has macje it possible to gain some iii?)<Mv^ant 
insights concerning the sensitiviHiy of^ the two instruments. The mcxst 
. important finding was that minutes of classroom teacher instruction seem 
to be related to studfeot performance when measured by the criterion- 
referenced tests but not when measured by the more traditioneUL standardized 
test. ! 



*Economics Dex>artuient, Indiant^ University. The work reported upon 
here was part of the results of a Joint effort by the author and Robert 
O'Reilly and Stephen Kidder of^he New York State Education Department. . 
Their assistance and cooperation is greatly acknowledged. 

Part of the work discussed in this paper was financed by the National- 
Conmission on ProductijVity and part by the Carnegie Corporation of Ney York. 
Some of the work was done while the author was a member of the senior re- ^ 
search staff at the Uriban Institute. * 



The Paracjlgmatlc Framevork ^ ^ 

13he paradigm that economists use for viewing productivity questions 

ifl the "production function", which involves description of the possible 

relationships of the ingredients to ^ production process to the level of 

output (or outputs), - It is virtualljy eJ.ways true that the inglrec^ients 

can be mixed in different ways to, obtain the same amount of product, and 

since the ingredients usually have different costs, some mixtures that 

produce a given product level will be less expensive ^than others, i.e., 

♦ 

they will be more efficient, rt 
I 

Hiis reasoning can be applied to a schooling process as well as to 
any other, andl^for reading instruction, for example, we cduld easily 
identify a nuntber of important ingredients: three or fbur^main types of 
teacher whole group instruction, , specialist instruction, assistance by 
paraprofes,sionals, etc. Since all these will probably hav.e different 
costs (some types of fnstruc4ion consume more time, remuneration varies " 
by type of ihstructor, etc.)i different combinations of ingredients will 
imply different costs. Thus, if we are to satisfactorily deal with pro- 
ductivity issues in reading instruc.tion, we need to obtain information 
on the relationship of the various instructional process inputs to stu- 
dent reading peiWormance . , All this assumes of course that a suitable 
laeasxure of reading performance is available. 

Since it is resource use that is our central concern within this 
pa!^digmatic framework, it is in^portant that the instructional inputs to 



i^i 



*This is true for all but the most trivial of production processes 
where a single input is used to "^produce a single output. 

For flurther discussion of the production function paradigm, see 
Bowles* (1970), or Kiesling (l971a). ^ Also any standard microeconomics 
textbook can consulted. ^ 
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be examined ^e quantified in a manner that is m^ningful in resoui^ce-use 
terms. The way to do this. that immediately suggests itself is to deter- 
mine the time spent on each activity (in minutes per week, say) since this 
can be imnediately converted'^to resource costs. (This assumes of course 
that we^^ssess a fairly accurate notion of^what is happening within ^ ^ 
elapsed time periods. ) 

study Objectives 

yhe analytical approach just discussed was used to pursue two main 
research objectives. The first was to gain insights into the efficiency 
of the various alternative "factors of production" whiph go into the 

" teaching of reading. Since this may vary according to type of student, 
part of this objective was to see whether apparent productivity 'patterns 

^ were similar for students from different socio-economic backgrounds. 

The second objective was to compare the apparent effectiveness of i the 
instructional inpXits when reading progress of the same students was measured 
by the criterion-referenced tests and the norm-referenced test. As most 
educational researchers are fully aware, the latter type of test for basic 
cognitive subjects has pr6vided the outcome measures for virtually of 
the "educational production function" work of the past. Much of this v(ork. 
has shown both poor and inconsistent relationships between schpol inputs 
and student performance, and many have wondered whether this may have 



*In the past few years several educational writers who are not econo- 
mists have also pointed out the value of quantifying school inputs in terms 
of time. See Bloom (197^), or Wiley and Harnischf eger (197^).' 

At least two empirical studies have been done which^ have quantified 
Inputs in this way. One is & $tudy by Conant (1973) who carefully traces 
the teaching productivity of teachers and paraprof essionals . The o.ther is 
a ^tudy of compensatory reading instruction by the author (Kieslin^, 1971b). 

**See Averch, al. (197U), pp. U6ff, 
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•^been due to imprecise measurement (vis-a-vis the objectives jof giyen 

school' districts) that may obtain when such tests are used, 

■ ^ - ' -. 

The Measurement Instruments 

The norm-referenced measurement instrument used In the study was the 
California Achievement Test, vocabulary And conrprehension sections. The 
CAT is a widely respected commercially available test. Secondly, a set 
of criterion-referenced tests (CRT's) were used which have^ been developed 
in the New York State Education Department over the past several years. 
The key difference between these tests ajid "standardized" tests (such as ^ 
the California) is that t'^st objectives are based upon the objectives- of 
a much more circumscribed set of schools. There were two main reading 
aspects covered: vocabulary and comprehension. For the former, words 
were selected from the basal readers used b^ participating school districts 
Comprehension objectives chosen were basfed in large part upon statements of 
teachers in the san5)le schools concerning their own priorities. Textual 
pass6iges for comprehension exercises on the tests were also based upon 
passages in basal readers. 

These tests also differj^d from norm-referenced tests in ^at tests 

for a given level contains material with a degree of difficulty that is 

applicable only to that level. (Levels w^re made to correspond to standard 

grfiuies, using basal readers for guidance.) Mastery of a level implies 

getting nearly all items correct at that level. Thes^ tests are much more 

homogeneous than norm-referenced tests, which jbave questions at widely . 

varying .difficulty levels^- many of which have no conceivable suitability 
^ \ 

* *The tests have been developed by the Bureau of School and Cultural 

Research, under the direction of Dr. Robert O'Reilly. 

I, ■ 



for measuring instruction in a glVen school grade. 

Another important difference between the two types of test is that, 
teachers ar« normally much more aware of the content of the criterion- 
referenced tests, not Only becausi they participate in specifying objiec- ' 
tives, but also because they are meant' to have access to the tests before 
instruction takes place, since the tests themselves are meant to be an ^ 
instructional aid. Teachers therefore are being constantly given timely 
feedback from prior testing. (The tests a!^ administered often; every 
two to three weeks and teachers aire given the results within five to ten^ 
school days.) Thus teaching to th^ tests is possible and teachers must' 
be cautioned against teaching to specific items if long-run instruction 
is to be effective. 

Finally, it is only fair to state that these tests did not have one 
attribute that would have been desirable, which would be to meas\a*e only 
material taught in school and therefore where correct answers cannot be j 
given based only upon intelligence. While it may or^^^nay not be possible 
to constryict such a reading test, good scores on these criterion-referenced 
tests- cannot be divorced from the advantages of native intelligence^ 
although it is reasoqable to expect that the intelligence component of 
the performance on these tests .is less than would be the^case for norm- 
referenced tests. 



*By all indications there has been little trouble with this in this 
New York state program, which is probably because particlj)ating teachers 
have been confident thg^t they will never be rated in an^T way based on 
test results. 

**For further discussion of the tests, see Athey (1975) f^nd a'Reilly 
(1973): 



The Data Set 

The data were gathered for approximately 5800 Uth, 5th, and 6th grade 
students i/n five New York state school districts duriag the spring semester 
of the^l91^U-75 school year. Four of the five school districts were urban 



.^^ore-citi^i 



IS, while the^fifth district was suburban. The choice of dis- 



tricts waj predfcated largely on the fact that teachers in these districts 
were in substantial Agreement concei^ning their teaching objectives, althpugh 
the districts (and schools within them) varied in socio-economic character- 
istics. Students were administered the California Achievement Test in 
February and in May, and were also given four or five test administrations 
of the proper level 9f the CRT during the course of the semester, with 
test level being determined by the highest level on which the student 
missed a substantial number of questions. \ 

Other than test scores, the data were gathered using taped interviews 
with principals, teachers, specieULists, and selected teacher aides. To 
meaningfully pinpoint instructional resources going into reading, teachers 
fiUfid specialists were asked to supply minutes per week of reading instruction 
in each of fotir instructional modes: whole group,, small group, individualized 



Each test level corresponded roughly to a standard grade and each 
level had five parallel test forms each of which having between 30 and UO . 
items. 

To choose a student's proper beginning level, the teacher was asked 
to make an estimate and the test for that level was administered. If / 
that level turned out to be incorrect, the proper level was then readmJ^n- 
istered and became the beginning score. If a student scored medium %o 
high on one level, he may have changed levels dxiring the semester, but 
for comparability we have not included results from such level changes. 
Not many observations fell in that category, largely because each level 
spans an academic year and the instruction spanned les^ than haULf of an 
acaxiemic year. 



7.' • 



7 



instruction, and individual help.* An important virtue of the data set 

put together in this way was that It was possible to obtain estimates for 

each individual child » When possible, estimates of one instructor were 

cross-checked against those of another. Project personnel were mature 

employees of the Bureau of School and Cultural Research and of the Urban 

Institute who were given careful training concerning interviewing techniques, 

Teacher interviews typicaill^ lasted one hour. 

A number of other questions were asked in the teacher questionnaire 

and some data were gathered from- school records . Such information included 

student age, absentee rate, father education, emd father occupation. 

While project personnel did their best to get accurate information on the 

4 

latter two variables from teacher, princii)al, and school records, it was 
' stiU^/^jnly possible to^tain it foi^ about sixty percent of the students. 
Also gathered was considerable information concerning use of audio-visual 
and other instructional materials. 

Finally, project personnel were not able to maintain desirable 
quality controls for the data collection effort for two schools (120 
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*These terms were used as foUqys. During whole group instruction 
the entire class is involved at the same time, listening to instruction, 
recitation, questions^ etc. When sub-groups of the class are doing dif- 
ferent tasks at the same time, the instruction was termed "small group." 
Individualized instruction involved diagnosis of individual needs and 
instruction tailored for those needs, while individual help involved 
drill-skill exercises with individual students. 

**We decided to undertake such a difficult exercise only after some 
hesitation, but it did not turn out to be quite as difficult as it at, 
first appeared. The method used was as follows. We presented a teacher 
with a class list of his or her children and asked first "What instruction 
did all receive together." Next we asked if^there were any children whose 
type or time of instruction was different. If so, we then asked^ the teacher 
to name these students and then to supply* the applicable information. If 
other instructors were involved, we then proceeded to ask the other in- ^ 
structor about the type and amount. of instruction given each child by name. • 
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children) i-n one school district and these data were discarded • 



The Explanatory Model 

The statistical tool used for the explanatory models was single-sta^e 
multiple regression analysis. The variables included in the models, as 
well as the definition of the variables themselves, was motivated by the 
Research objectives above • In order to i'Yiquire into the productivity ques- 
t^ions posed, e^ight resoxirce -using variables were included. Seven of these 
were the ^st important ins true tione^l time variables (on both a priori ^ 
grounds and in terms of having meaningful variation in the data). The 
eighth was a. variable constructed to denote* the amount ^^jsI^ materials and 
equipment that were used in the instruction. 

Perhaps the most important, variable which must be accou^ed for in 
any determination of student post-test scores is their prej^test score. 
This helps to control for ability differences as well as for differences 
in prior preparation. Also, using pre-test scores as an explanatory 
variable, instead of using outright gains (subtracting the pre-test from 
the post-test score) allows for such systematic biases as the "regression 
to the mecui"* phenomenon to be dealt with by the model. 



*These were the only two schools representing a larger urbeui school 
district, and therefore only fovir school districts are represented in 
the study. ^ 

**Teachers were asked whether they used such equipment and materials 
as the following either occasionally or regtilarly: films trips, Slides^ 
overhead projectors, moving picture projectors, self -creativity materials, 
transparencies , charts , magnetic tapes, etc. Hie variable coun^jgd the 
niimber of positive responses to ik such items, with the "reguj.a^y" re- 
sponse being weighted as double the "occasionally" response. 

***However, the author has found in previous einpirical work that find- 
ings using both approaches are highly similar. See Kiesling (1971), 
Appendix A, for a complete discussion. 
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^ It would be reasonable to expect on a priori grounds that student 
age would be related to reading performance, and this is even woi^e true 
in the scusple used in this study since students could be placed tin a higher 
,or lower CRT level according to their accomplishments, it was also possible 
to gather information concerning absentee rates, and this variable is in- 
eluded for obvious reasons. 

Almost all educational production function models that have been used 
in the literature include a variable to account for the effects of student 
socio-eponomic background. The rationale for do^ng this is quite straight- 
forward: parental influences are important for motivation towards learning 
as well as for how muciT^ earning taJies place in the home directly. Since 
we were able to collect information on father occupation, it would seem 
sensible theref oi^e ^hat this be included in the explanatory model. How- 
ever, there are some qualifications that should be made concerning the 
use of this variable. 

F^rst, we must remember that we already have a v9.riable for pre-test 
score, and this can be expected to account for some of these home background 
effects. Secondly, it is true that single-stage multiple regression models 
of the educational process will be mis -specified no matter whether a SES 
variable is included or excluded. A third' qualification comes because in 
this study it was inqpossible for us to obtain data for more than one»-third 



*This is becQ.use SES is usually positively correlated with both school 
qufiility on the one haiid, and pupil ability, motivation, and home learning 
on the .other . Including the variabj^e gives the fdinily s^e of the credit 
that twB school should get, while excluding it gives school variables too 
much credit. The truth lies somewhere in between. A properly specified 
model will use a simultaneous equation approach which explains both aspects 
with separate explanatory variables, but the data requirements for doing 
this are considerably richer than was available in the present study. 
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of th6 students, and we ar6 not convinced that the one-third not collected 
are distributed r^domly. As it turns out, introduction of the father 
occupation variable does not affect the findings very much, and we there- 
fore use equations without the variable in a number of places. 

Finally, an atteii?)t was also' mad€?. to examine socio-economic differences 
by stratifying the children according to whether they attended low and non- 
low socio-economic status schools. 

Findings * 

Table 1 contains the principal set of fitted multiple-regression 
equations for all students (with the exception of %he 120 childr^ noted 
above.) ^ For reasons already explained, regressions are given with the 
father occupation variable both included and excluded. While there were 
six CPT levels used, resxilts are presented for only five of them, since 
^here were very few observations for students in the highest level (level 
7). Levels U, 5, and 6 are "proper*' for the children in the study sajnple, 
since they were in grades U, 5, and 6. Thus those' children in levels 2 
and 3 can be considered as being students who are below avereige in read^g 
attainment. 

The most important finding to be derived from the information in ^|[^ 
Table 1 is that student performemce as measured on the norm-referenced 
test is much more poorly related to the instructional inputs than is such 
performance when measured by the criterion-referenced tests. Only one 
input shows signs of a i)Ositive relationship to CAT scores, (classro^^ 
teacher small group instruction) and even that does not reach statistical 



^Means and standard deviations appe€ur in Table 6. 
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Results from Multiple Regression Equations Explaining 
V Student Reading Performance in Grades U,^ 5, 6, with 
and without Father Occupation Contuolled*' 
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Elxplanatory 
Variables 

1. Classroom Teacher: 
Whole Group 



CRT Level ? 
.0031 .0009 



CET Level 3 
.0091* .0059* 



CRT Level U 

.0005 .OOOU 





Instruction 


(.167) 


(.558) 


(.000) 


(.005) 


(.759) 


(.fil2) 


p. 


Classroom Teacher: 
Small Group 
1 ns truction 


.00U5» 
(.075) 


.0028* 

(.018) 


,0029» 

(.058) 


.0023* 
(.059) 


.0021* 
(.091) 


.0021 
( .OI4O) 

•1 




Classroom Teacher: 

Individualized 

Instruction 


.0056 
(.5U3) 


.0100 
(.11.7) 


.0106 

(.163) 


• 

• .0088* 
( .062) 


.0017 
(.696) 


.0022 
(.386) 




Specialist: 
Small Group 
Instruction 


.0087 
(.567) 


-.OO3U 
(.725) 


.0066 
(.502) 


.0132 
(•121) 


.0361 , 
( .115) 


.01I49 
(.267) 


^. 


Specialist: 

Individualized; 

Instruction 


.0017 
(.5J2) 


-.0021 
(.326) 


-.0019 
(.535) 


-.0027 
(.293) 


-.0079 
(.023) 


-.0088 
(.003) 


6. 


Paid Aide: 
Small Group 
.Instruction ^ 


-.0012 
(.928) 


.0017 
(.815) 


-.601U 

(.756) 


-.0025 
( .325) 


-.3UO 
( .001) 


-.02U1 
( .001) 


7. 


Individual Help 


-.0017 
(.773) 


.0007 
(.890) 


-.ooi3 ■ 
(.839) 


-.0005 « 
(.923) 


-.0058 

(.383) ' 


-.0086 
(.059) 


ft. 


Material and 

Equipment 

Index 


.OU95- 
(.762) 


-.llt4A 

(.311) 


-.2113 
(•059) 


-.1926 
(.021) 


-.3ir29\ 
( .000) 


-.3597 
( .000) 


9. 


Student Age 


.0320 


,.0115 


-.171U 


-.1517 

{ 088 ■) 


■.0185 


- .1216 


10. 


Number of Days 
Absent 


-.0371* 
(.059) 


-.0210 
(.1U2). 


.0031 
(.838) 


. 0015 ■ 
(.905) . 


.019U 
(.168) 


.0067 
(.52U) 


11. 


Father Occupation 

i 


-.0071 
(.98U) 




.3559* 
(•057) 




.1357 
(.355) 




12. 


Pre-test Scojre 


.708* 
(.000) 


.6o8» 
( .000) 


:583* 
( .000) 


.5U3* 

( .000) 


.733* - 
( .000) 


.726* 
( .000) 




Number of 
Observations 


153 


260 


320 


507> 


U38 


658 






.5U2 


.U29 


.363 • 


■ .30U 


.599 


.579 
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Expleuiatory ^ 
variables ; . 

1. Classi'oom Teacher: 
Whole Group 
Insttuctior^ • 

2. Classroom ^Teacher : 
, Small Group 

InstructioQ 



3. Classroom Teacher: 
.-Individualized 
Instruction 

U. Speci^ist; 

Small Group » 
Inst^ruction 

5. Specialist: ^ 
Individualized 
Instruction 

6. Paid Aide: 
Small ftroup 
Instruction ' 

7. Individual Help 



8. Material and 
Equipment 
Index 

9. Student Age 

10. Number c5f Days 
Absent 

11 . Father Occupation 

12. Pre-test Score' 



Number of 
Observations 



r2 



Table 1 ' 
(continued) 



CRT Level 5 



.0013 .0033 
(.576) (.129) 



.ooie -.0012 

(.529) ,(.U35) 
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,0010 -.0009 
(.ffSU) (.806.) 



.oiUi -.01^1 
(.U92) ' (.U12) 



.0055 -.0025 

(.7U3) (.-785) 

-.OOUl -.0088 
(.790), ■ ,(.1U2) 

.0185 .0125 

(.198) {.^hk") 

-.U066/ -.3061 

(.006) (.008) 

.-.1920 -.1227 

(.115) (.25U) 

-.0088 -.0253* 

(.6I15) rj^099) 

.5228* 
(.OOU) 

.713* .716* 

(.000) (.000) 



li36 

.61U 



582 
.558 



CRT Level 6 



.0036 .902U 
.109) (-.236^ 



CAT 



-.0008 -.OOOU 
(.738) (.96U) 



.ooU,o» .0035* .0019 • .0017 
.050) (.051) (.267) ' (.209) ■ 



.0071* .0057 ; .0012 -.0038, 
.088) (.128)' (.816)- (.288) 



.7U97 '-.0065 - -.0UU6 -.0355 
.OlU) (.025) (.011) (.002) 



.007U* .0752* 

.01?) (.012) 



.oUUo .051^* 
.137) , .(.097) 

.0626* .OU6O 
.099) (.191) 



.U206 
.000) 

.2087 
.iU5) 

.0137 
.353) 

.2538 
.103) 

.713* 
.000) 



-.2771 
( .010) 

.1671 

(.217) 

.0027 
(.8U7) 



.712* 

( .000). 



397 ' U72 
.67U .623 



-.120 -.0123 
(.017) (.061) 



-.0213 ^-.0119 

(.015) ^'fi025) 

-.019U -.0126 

(.016) (.030) 



.1585 ■ 
»(.l66) 

.0555 
(.630) 

-.0528* 
(.012) 

1.507* 
(.000) 

.863* 

(.000) 



2231 
.790 



. .0560 

(.520) 

^-.1085 
(.232) 

-.0635* 
(.000) 



.905* 

(.000) 



3172 

.797- 



TlEible notes appear <after Table 5. 
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Table 3 

Effect on Main Instructional Variances o^ Including 
Dummy Variables for School District 



Explanatory 
Variable 



CRT Level 2 



CRT Level 3 



1. Classroom Teacher: .0009 , .0011 .0059* .0066* 

Whole Group (.538) (.559) (.005) (.002) 

Instruction 



CRT Level 4 

.0004 .0019 
(.812) (.232) 



Classroom Teacher: .0028* .0019 . .0023* .0026* 

Small Group (.018) (.353) (.059) (.OAA) 

Instruction * 

Classroom Teacher: .OlpO .0078 .0088* .0082* 

Individualized , (.1A7) (.256) (.062) (.0807) 

Instruction ^' 



.0021* . .0025* 
(.OAO) (.026) » 



.0022 .0008 
(.386), (.775) 



) 



A. Specialist: 

Individualized 
Instruction 



-.0021 , -.003A -.0027 -.00A5 -.0088 -.0103 
(.326) (.125) (.^93) (.b82) (.003) (.000) 



5. Dummy Variable: 
District A 

6. Dummy Variable: 
District B 

7. Dummy Variable: 
District C 



0.L91 
(.812) 

rO.732 
(.A28) 

-2.875* 
(.007) 



-0.238 
(.633) 

-2.681* 
(.000) 

-2.217* 
(.002) 



0.316 
(.A61) 

-1.694? 
(.000) " 

-2.212* 
(.000) 



Table Notes Appe^ After Table 5 
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Explanatory 
Variable 



( 



1. Classroom Teaqher: 
VAiole Group 
Inat ruction 

2» Classroom Teacher: 
Small Group 
Instruction 

3. Classroom Teacher: 
Individualized 
Instruction 

A. Specialist: 

Individualized 
Instruction 

5. Dummy Variable: 
District A " 

6. Dummy Variable: 
District B 

7. Dummy Variable: 
District C 



Table 3 
(continued) 



CRT Level 5 

.0033 .0025 
(.129) (.238) 

w 

^0012 -.0006) 
(.435) (.696) 



-.0009 
(.806) 



-.0121 
(.A12) 



-.0083 
(.021) 



-0.608 
(.282) 

-1.329* 
(.022) 

-4.791* 
(.000) 



V 



CRT Level 6 



.0024 
(.236) 



.0014 
(.454) ^ 



-.00004 -.0031 
(.984) (.092) 



.0035* .0039* .0017 -.0003 
(.051) (.024) ^ (.209) (.836) 



.p057 
(.128) 



.0008 
(.837) 



-.0151^ .0752* .0429 
(.090) (.012) (.121) 



-.0038 
(.288) 



-.0123 
(.001) 



-1.386* 
(.027) 

-3.708* 

(iOOD) 

-6.177* 
(.000) 



-.0007 
(.851) 



-.0085 
(.018) 



0.563- 
(.#2) 

6.2i7* 
(.000) 

4.451* 
C.OOO) 



\ 



. Table 4 l6 
Comparative Results for Students in Low and Non-Low SES Schools 



Explanatory 



CRT Level 2 



CRT Level 3 



CRT Ldvel 4 





Variable 


Low 


Non-Low 


Low 


Non-Low 


Low 


Non-L»w 


% 


1. 


Classroom Teacher: 
'Whole Group 
Instruction 


. 0022 
(.261) 


a. 


.0050* 
(.043) 


.0052 
(.292) 


.0024 
(.(^44) 


-.0063 
^043) 


• I 


< 2. Classroofn Teacher: 
^/^^mall TJroup ' 
Instruction 


/way i\ 

.0)019 
(.n351) 




.0017 
(.256) 


.0005 
(.875) 


.0041* 
(.003> 


-.0012 
(.600) 


> 




Classroom Teacher: 
Individualized' • 
Instruction 


.0378* 
,(.003) 




.0067 
(.459) 


.0024* 
(.006) 


.0035 
(.250) 


.0039 
(.488> 


► 


4. 


Specialist: 
Small Group 
Instruction 


-.0095 
(.370) 


« 


.0033 
(.759) 


.0064 
(.644) 


.0088 
(.596) 


.0284 
(.232) 


J 


5. 


Specialist: 
Individualized ^ 
Instruction 


-.001*1* 
(.183) 




-.0008 
(.#38). 


-.0260' 
(.004) 


-.0129 
(.014) 


-.0246 
(.245) 




6. 


Pai|d Atde: 
Small- Group 
Instruction 


.0121 
(.351) 




.0011 
(.6^67) 


♦ 

-.0157 
(.205) 


-.0231 
(.001) 


.0956 
(.383) 




7. 


Individual Help 


-.0016 


y 


.0044 
(.540)^ , 


-.0079 
(.257) 


-.0027 
( .603) 


.0035 
(.848) 




8^ 


Materials and 
Equipment Index 


-.822 




.0097 

( .952) 
* 


.2123 
(.395) . 


-.231 
(.022) 


-.0459 
(.769) 




9. 


Student Age 


-.0233 
(.857) 




-.124 
1 (.286) 


-ao67 

(.506) 


-:233 
(.005) 


-.1123 
(.369) 




10. 


Number of Days 
Absent 


-.0233 
(.243) 


« 


-.0027 
(.890) 


-.020§ 
(.550) 


-.0022 
(.862)' 


.0636 
(.011) 




11. 


Schools: Mixed 
Black and White 
(dummy variable) 


(.515) 




1.017* 
(.094) 


0.913 
(.476) 


1.533* 
(.002) 


-2.807 
(.201) 




12. 


Schools : 
Mostly Black 
(dummy variable) 


3.288* 
1.018) 




-1.157 
(.177) ' 


(.000) 


-.395 
(.455) 






L3. 


Pre-Test Score 


.471* 
(.000) 


■ 


. 395* 
(.000) 


(.000) 


♦ 

7 ^A* 

(.000) 






Number of 
Observatic^ns 


174 




275 


179 


386 


239 








.387 




.230 


.338 


.547 


.526 


ERLC 
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Table Notes appear 


after Table 
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Explanatory 



CRT Level 5 



CRT. Level 6 



CAT 



ERIC 



Variable 


Lov 




Low 






won—uow 


1. Claasroom 'Teacher: 

jf^Jt/tiole Group 
'. Instruction 


.0022 
(.41A) 


.0018 
(.637) 


, .OOAl 
( . 300) 


-.0017 
(.682) 


.OOIA 
(.510) 


.0021 
(.587) 


2. 


Classroom Teacher: 
Snurll Group 
Instruction 


.0011 
(.607) 


-.0041 
(.152) 


.0068* 
(.019> 


.0066* 
(.038) 


.0043* 
(.019) 


-.0011 
(.672) 


3. 


Classroom Teacher: 

Individualized 

Instruction 


.00005 
(.99A) 


-.0109 
(.Q72) 


.0034 
(.589) 


.605A 
(.540) 


.0063 
(.198) 


-.0201 
(.000) 


A. 


Specialist: 
Small Group 
Instruction 


-.013? 
(.380) 


-.0021 
(.951) 


-.383 
(.335) 


b. 


-.0221 
(.063) 


-.0A13 
(.081) 


5. 


Specialist: 
Individualized . 
Instruction^ 


-.0070 
( Left's 

\ . HMD/ 


b. 


• oatV 

( 252lS 


.1097* 

( 008^'' 


-.0051 , 

( 7AS^ 
V . *'»J/ 


-.0127 


6. 


Paid Aide: 
Small Group 
Instruction 


(.26A) 


_ ni7i 
(.405) 




(.121) 


(.048) 


. UUDo 

(.767) 


7. 


Individual Help 


.0009 
(.958) 


.0173 
(.389) 


-.0585 
(.365) 


.1032* 
(.031) 


-.0108 
(.130) 


-.0221 
(.016X 


8. 


Materials and 
Equipment Index 


-.3A9 
(.OAO) 


-.489^ . 
(.035) 


.1602 
(.380) 


-.2A65 
(.263) 


-.2702 
(.100) 


.1826 
(.339) 


9. 


\ 

Student Age 


(.380) 


(.2*23) 


(.023) 


1 1 Q7 

(.521) 


(.320) 


(.798) 


10. 


Number of Days 
Absent 


-.0321 


-.0183 
( SS8) 


-.0278 


-.0325 


-.0A26*^ 


\ \006A 


11. 


Schopls : Mixed 
Black and White 
(dumny variable) 


1.1513* 
(^016) 


1.288 
( 233) 


2.637* 
(.003) 


0.282 
(.875) 


-4.312* 

( 000> 


5.155* 

( 000') 


12. 


Schools: 
Mostly Black 
(dummy variable 


2.A9A* 
(.012) 




2.080* 
(.023) 




-1.A51 
(.070) 






Pre-Test Score 


.670* 
(.000) 


.659* 

(.000) 


.58A* 

(.000) 


761* 
(.000) 


• Off 

(.000) 


(.000) 




Number of 
Observations 


288 


276 


190 


248 


1A58 


i 

1443 




r2 • , 


.582 


.536 

18 


.622 

> 


.708 


koi 


.780 



y 



• Table 5 
Results for Type of Instructor: 
Students In Low and Ndn-Lqy SES Schools 
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Minutes per week 
Instrtllfitlon by: 

Class room Teacher 
Reading Specialist 



Paid Paraprofessional 
Aide 

Unpaid Teacher Aides 



CRT Level 2 
Low Non-'Low 

.'^^11* a. 
( ,096) 

-.0058 • 
(.066) 

.0032 
(.548) 

-.0131 
(.154) 



CRT Level 3 
Low Non-Low 

.0023* .0024 
(.093) (.355) 

-.00004 -.0109 
(.992) (.078) 

.0011 -.0095 
(.665) , (.074) 

-.ooir-<.-.oo7i 

(.538) (.657) 



. CRf Level 4 
Low Non-Low 

.002-4* -.0003 
(.035) (.894) 



-.0053 
(.199) 

-.0111 
(.006) 

-.0057 
(.866) 



.0032 
(.844) 

.0052 
(.813) 

-.0646 
(.016) 



TABLE NOTES 



The figure In parentheses under the coef flcent of net regression denotes the 
level of probability with which it could occur that the result occured by chance. 
A figure of .050 means the probability is 5 chances in 100 for example. Coefficients 
significant at the ten-percent level or lower and which have the hyppthetically 
correct sign are marked with an asterisk. ^ ' ' ' ■ , 

a. There were too few observations in the population, for this regression 
equation to be meaningful. • 

b. No observation had a non-zero value for this variable. 
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Table 5 
(continued) 



Minutes per week CRT Level 5 CRT Level 6 CAT 



Instruction by: 


Low 


Non-Low 


Low 


Non-Low 


Low 


Non-Low 


Classroom Teacher 


.0026 
(:168) 


-.0044 
(.068) 


, .0050* 
(.017) 


.0054* 
(.068) 

.1132* 
(.007) 


.0040* 
(.009) 


.0011 
(.692) 


Reading Specialist 


(.522) 


-.0032 
(.928) 


.0015 
(.229) 


-.0107 
(.008) 


-.0260 
(.019) 


faid Paraprofesslonal 
Aide 


. -.0051 
(.393) 


-.0046 
(.746) 


' • b. 


.0430 
(.142) ' 


-.0106 
(.013) 


-.0194 
(.066) 


Unpaid Teacher Aides 


. b. 


i b. 


b. 


.0874* 
(.083) 


-.0172 
(.345) 


-.0077 
(.731) 



r 

I 

I 
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Table 
Description of 



6 * 

Variables 
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Variable • . 

Whole Group Instruction, , 
Classroom teacher, Mlmites Per 
Week (MPW) 

Small Group Instruction, Classroom 
Teacher, MPW 

Small Group Insti^uction, Specialists, MPW 

Small Group Instruction, Paid Aides, .MPW 

Indi^dualized Instruction, Classroom 
Teacher, MPW 

Individualized Instruction, Specialists, 
MPW 

Whole Group Instruction, Total MPW 

Small Groi;u> Instruction, Total MPW 

Individualized Instruction, Total MPW 

Individual Help, Total MPW ^ 

Classroom Teacher Ins true tl^. Total MPW 

Specialist Instruction, Total MPW 

Paid Aide Instruction, Total MPW 

Unpaid Aide Instruction, Total MPW 

Criterion Referenced Test, (CRT> 
Level 2, Test 1 (February) 

CRT Level 2, Test 4 (May) 

CRT Level 3, Test 1 

CRT Level 3, Test 4 

CRT Level 4, Test 1 

CRT Level 4, Test k . 



m 

• ^4.07 

166.35 
2.16 
4.8^ 

11.50 

11.91 
64.07 
173.65 
24.22 
7.fl5 
244.31 
14.32 
9.75 
1.42' 

12.72 
13.22 
13.53 

!" 

f09 
16.68 



1 

16/C 



102.59 

140.79 
15.79 
33.85 

51.07 

50.61 

< 

102.59 
146.88 
72.82 
' 30.16 
146.18 
54.07 
43.60 
12.22 

5.05 
5.05 
4.12 
4.45 
4.81 
5.23 



c 



ERIC 
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Table 6 (continued) 

Variable 
CRT level 5. Test 1 

CRT Level 5, Teat 4 / - - 

CRT Level 6, Test 1 

CRT Level 6, Test 4 

California Achievement Test, 
Pre-Test (Febniary, In Mpnths) 

California Achievement Test, 
Post-Test (May, In Months) 

Index of Materials <Used . 

Student A^ (Half Years) 

Number of Days Absent, Student ^ 

Father Occupation Index 



m 


8 


18.07 


5.99 


18.60 


6.24 


18.59 


6.01 


19.01 


5.95 


50.18 


21.39 


55.06 


' 

22.09 


3.A9 


2.10 


21.83 


2.04 


10.75 


11.51 


3;48' 


1.24 
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significance at .the twenty percent level. Indeed, most of the instructional 
inputs are negatively ^•elated to student performance on the CAT and this 
^ften occurs at advanced levels of statistical significance. This is 
expecially fcrue^ for non-classroom teacher instruction. 

The positive relationship ot school instructional inputs to the CRT 
scores is especially noteworthy for instruction by ^the classroom teacher, 
particularly when it is in the small group mode. However, specialist 
instruction shows signs of being important in levels U And 6; while in 
level 5 no input seems to be related to student performance. 

There is an important reservation that must be made concerning the 
relationsh^s to these ctiteribn-refer^enced test findings however, which 
is not directly shown in Table 1. Although many instructional inputs 
were significantly related to student performance measured in the CRT's, 

it is also true that the amount of varia^tion explained was not large. 

t, /• 

Thus, more than 300 minutes of classroom teacher time would be required 
for a one point gain in CRT score. This is about three standard deviations 
of the teacher whole class instruction variable. For small group instruc- 
tion more theufi UOO minutes woiild have been required, again about three 
standard deviations. (There were typically 30 items per CRT level, 
corresponding to an academic year.) By contrast, apparent pupil average 
gain as measured by the standardized measure was more than .U grade 
equivalents. Can the difference be explained as due to the difference 
of the quantity of home learning as opposed to school induced learning? 

Despite our theoretical expectations about the influence of a variable 
for parents' occupation, its inclusion in the explanatory equation does not 
seem to alter the findings very much, and since more than one-third of the 
observations have .this information missing, the variable is not included 

23 
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in the fitted equations presented in the rest of the paper. However, it 
is quite noteworthy that father's occupation displays a much stronger rela- 
^tionship to performance on the CAt than on the CRT's, which suggests 
the latter are more sensitive to learning originating in the school as 
opposed to that in the hbme, just as we would expect a priori , This would 
cvei^be more true of course for. learning outside the proper class level , 
which is captured by standardized tests and (correctly, from the standpoint 
of studying schools) not captured by the CRT*s. 

One finding contained in Table 1 is both puzzling and potentially 
importMt, eui^r^fliiat is the consistently negative relationship of the amount 
of materials and equipment used to student performance as measured b^the 
CRT's.. A. partial explanation might be the fact that the lower SES schools 
have more equipment, perhaps purchased with Title I money. Otherwise, 
the use of equipment euid materials would seem to be counterproductive for 
some mysterious reason. The story becomes even more perplexing when we 
notice that for student progress as measured by the California Achievement 
Test the materials and equipment variable reverses sign and, when father 
occupation is controlled, even approaches statistical significaince. 

Another result shown in the table which is of interest has to do with 
the marked differences between the relationships of the instructional in- 
puts to student performance in CRT levels 5 and 6. One possible explanation 
for this is that level 5 has more than its share of older students with 
low ability. Evidence to support this is contained in the coefficients 
for student age in the two levels, positive in level 6 and negative in 
level 5* 



ERLC 



*The Index value for low SES schools was 3 -92 and for non-low schools 

'3.59. 
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Finally, som^ explanation is required for the number of statistically 
significant negative relationships for the specialist, paid aide, and 
individual help instructional inputs. A very likely explanation' for this 
is that such instruction is generally devoted to "remediation" for low 
achieving students. This is especially likely in this data set because 
of the low amounts of instruction^ time that were devoted to these modes. 

It appears in Table 1 that regular instruction by the classroom teacher 
is more important for student performance than for instruction by other 
types of instructional personnel. This is confirmed in Table 2, where re- 
sults are given for the same model as in Table 1 (without the father 
occupation variable) except that minutes of all type instruction by type 
>of instructor is used. Excep^fqr level 5, the only significant or near- 
significant net regression coefficients are for the classroom teacher. 
These findings may be due in part to the strpng possibility mentioned 
abovGjprthat other types of instructors were used for lower ability students. ; 

It was also possible to test whether there were meaningful differences 
between type of instruction: whole-group versus small-group, etc. There 
*were none, and these findings are omitted. 

In our sample there were only fotxr usable school districts, and with 
tfsuch a small number it is perhaps not very meaningful to inquire into 
school district effects. However, it is possible to at least ascertain 
whether school district per se seems to be an' important variable by merely 
including dummy variables for each district. The results when this is done, 
for the main instructional variables, are present*»<i in Table 3. The 
dummy variables show significant differences by school district more often 




*Sec the means presented in Table 6. 



I 

V , 

than not, but, except for at CRT level 2, including the dummy variables 
does not affect the explanatory power of the instructional variables. 

Perhaps the most intriguing result in Table 3 is ^the rather spectacu- 
lar reversal in sign and significance for the dummy variables for districts 
B and 0 when reading performance is measured by the CAT. No ready explana- 
tion for this comes to mind, but again it happens that results for the two 
types of test are highly divergent. 

The last hypothesis tested in the study had to do with whether instruc- 
* tional results differ for high and low SES pupils and whether results seemed 
to be much different when there were large percentages of black children 
in a given school. To do this each school was placed in a "low-SES" or 
"non-lo^^S" category, and schools were also classified as to whether 
all-whit^, all-black, or racially mixed. Then the, regression explanatory 
model was fitted to the performance of children in the low and non-low 
SES groupings,' and within each of these categories dummy variables were 
introduced to account for racial mixtiare. ^ These findings are presented 
in Tables U and 5. The hypothesis that instructional relationships differ 
for the low cuid non-low SES schools is confirmed by the information in 
these tables, with instruction being more closely related to the gains 
' of low SES pupils. This even holds true with respect to performance as 
measured by the California Achievement Test, which is remarkable in that 
it is the only instance in the entire study where this is true. This close 
relationship for the low SES students may reflect the fact that due to the 
nature of their homes a much greater percentage of their performance gains 
are due to school lnst;,ruction than is true for the high SES children. 

Finally, while the results of race in the findings in Table h seem 
somewhat mixed, in general the all -black schools normally show up as 

2G 
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performing better onigHie CRT tests and worse on the NHT* The inQ)lication 
of this relationship is that the poorer shoving for blades in many studies 

using norm-refeifenced tests is indeed due to their poorer home environ- 

. ... ^ 

mentSy and vhen this is properly ad;)usted fqr (or when an instrument that 
measures school effects is used) blacks do ae well or better than whites, - 
However these findings axe based on so few school obseirvations that it 
would be wrong to generalize them very far. 



Cost Ramifications 

It is^perilous to present cost ramifications (and therefore direct 
efficiency estimates) of -findings from>a single study where the effective 

M. 

instruction lasted only eight to ten weeks.'' However^ the delineation of 
the procedure for arriving at cost estimates should not be' without inter- ^ 
est, nor should a set of illustrative calculations be without useful 
information. 

* It is not very difficult to derive meaningful cost estimates. First 
the calculated caefficient of net regression is divided into 1.0. This 
shows how many minutes of instruction it will take to raise pupil| scores 
one point. This' is then divided by 138O, the figure taken is the total 
* number of . instructional minutes, in a teacher week. This figure is then 
multiplied by the instructor *J5 yearly salcury and divided by class size to 



*What would be needed before po][.icy decisions could be directly based 
upon such calculations would be (1) carefully controlled variation of in- 
puts, (2) good control for student socio-economic characteristics (3) studies 
si>anning a time perioji of at least a year and preferably longer, and (U) a 
number of replications. 

**This figure comes from a finding by Educational Turnl^y Systems (1971) 
that* teachers spend U.6 hours per day doing instruction. 
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yield the cost per student per point gain. The only Inputs that were 
cwislstently significant enough across most of the CRT levels to merit ^ 
making calculations were whole-group and small-group instruction by the 
classroom teacher. Taking average coefficients across all levels (and 
using equations both with and without father occupation), t^he figures ob- 
tained were $87 and $if31 cost per point gain per student for whole-group 
and^. small -group instruction respectively. For the CAT results, only two 
coefficients were positive, those for small group instruction €uid the 
materials index. Calculations for the cost for .1 grade equivGLLents of 
gain for these two Inputs came to $575 wid $502 respectively. 

Finally, the oijly student population where a fair number of inputs 
were positively related to reading performance was that in CRT level 6. 
Therefore the cost calculations for level 6 Inputs were cbmputed and are 
as follows: 

Teacher whole-group instruction $ 89 ^ 
Teacher small-group instruction $ 276l 
. Teacher lndivi^u€LL*lzed instruction $1132 
^Specialist small group instruction $ 117 
Paid aidfe smeQ-l group instruction $ 9 . 



*The median figures in our data for cla^s size and small group size 
were 27.0 and 7.O respectively. These were the figii^es used. $10^000 and 
$12,000 were the figures used for the annual salaries of classroom teachers 
and specialists respectively, paid aides were assumed to receive $5.00 per 
houTf including firinges. 

**Calculations for costing out the materials index are not difficult, 
merely somewhat tedlou^i. Needed are data concerning the cost of each itiem,^ 
its normal life, and how many minutes per week it is^used. This information 
can be used to construct a cost-weighted materials index if desired. The 
index used in the regressions was not weighted and therefore an average value 
for each item needed to be taken.. Assumptions were also made that regular 
use meant half time and occasional use one-fourth time^ a procedure which 
undoubtedly overstated the cost. However, results for the materials index 
were so indifferent that it was not considered worthwhile to spend the con- ' 
siderable time it would have taken to make its costing precise. 
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Obviously the use of para-professloncLLs, If theiy can be made to be 

w 

instructionally effective, is quite cost-effective. Otherwise, teacher 
whole-group instruction has th^ least cost. 

Concluding Cotanents , , ^^^^ 

The data, analysis in this study has yielded important findings with 
respect to three lines of inquiry. First, it was asked whether some school 
inputs Gore more effective in teaching reading than others. It was found 
that the activities of the regular classroom teacher seem to be consistent- 
ly more inportant, although other Inputs were sometimes related to perfor- 
mance, especially in CRT level 6. Next was aske;d whether the relationships 
of school Inputs to reading performance as measured by criterion-referenced 
tests Alight not be different from those measured by norm-referenced tests. 
It was found that the former measure was related to school inputs much more 
consistently. Finally, we asked whether input-performance relationships 
might not be different for students from different •socio-economic back- 
grounds. When students were. separated according to whether they attended 
low or non-low socio-economic status schools, it wa(B found thiit school 
Inputs were moi^e consistently related to low SES ^tudent performance. 

While it wotlld be dangerous to conclude too much from the findings of 
a single study which monitored only about nine or ten weeks of actual in- 
struciion, these findings are still qui\e suggestive. After all the 
non-findings that have been turning up in^^e enpirical multi-variate 
literature concerning the effect of the classroom teacher, it is reassuring 
to see some more positive results using tests which are perhaps more sensi- 
tive than those used in that literature. Itowever, even though the ifistruc- 

\±on period was quite short, the average student gains on the criterion- 

«» 

referenced tests are surprisingly low, only about one point on, a 
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thirty-polnt Instrument (buUt to ccnrer one school yeajr). On the other 
handy students gained acre than four-tenths of a year measured in standard- 
ized grade equivalents. Could we have succeeded^in distinguishing between 
home effects and school effects? Or at least ve nay have distinguished 
between Instruction proper to t!he grade level and instruction that may have 
somehow given the students skills which are "proper" for other grcule levels. 

Finally, we must make an important reservation about the lack of poB±^ 
tive relationships to reading performance of instruction by specialists and * 
para-professionals. Two things can be said about this. First » it is a fact 
that in this smmple of classrooms non-^classroom teacher instruction w^ 
siiiq;>ly not used very much.* It can be argued that instruction will only 
begin to take hold after it reaches some threshhold, and that for almost 
all the cases in thi^s saoiile it is difficult to see how such a threshhold 
could hkve been readied. Secondly » one strongly suspects that instruction 
by non-classroom teachers is typically used for children who are "trouble- 
some"; who have traits both in lack of intelligence and motivation which 
could not have been controlled for by the relatively crude non- instruction 
variables used in this study. Therefore no conclu8ionA^>about the effective- 
ness of the inputs of such personnel should be made until such time ^s their 
effectiveness can be examined using more controlled experiments. 



*See Table 6r. Specialist small group instruction averaged only two 
minutes per week, paid-aide small group instruction ^ive minutes per week, 
and specialist individual Instruction 12 ndnutes per wedi. Of .the ^88? stu- 
dents in the .total sample 9 ^717 had no small group sj^ecialist instruction^ 
^702 no small group paid aide instruction^ ^3^0 had no specialist individual 
ized instruction, and ^830 had no paid aide indiviclualized instruction. 

**a9iis statement is reinforced by the fact that the ^author found strong » 
positive relationships between specialist instruction and reading gains by 
Title I children in an earlier study of California Title I projects. 'See 
Kiesling (1971b, 1972). 
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